Community-driven data grids

E-science communities and especially the astronomy community have put tremendous efforts into providing global access to their distributed scientific data sets to foster vivid data and knowledge sharing within their scientific federations. Beyond already existing huge data volumes, the collaborative researchers face major challenges in managing the anticipated data deluge of forthcoming projects with expected data rates of several terabytes a day, such as the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS), the Large Synoptic Survey Telescope (LSST), or the Low Frequency Array (LOFAR). In this thesis, we describe and investigate community-driven data grids as an e-science data management solution. Community-driven data grids target at domain-specific federations and provide a scalable, distributed, and collaborative data management. Our infrastructure optimizes the overall query throughput by employing dominant data characteristics (e. g., data skew) and query patterns. By combining well-established techniques for data partitioning and replication with Peer-to-Peer (P2P) technologies, we can address several challenging problems: data load balancing, efficient data dissemination and query processing, handling of query hot spots, and the adaption to short-term query bursts as well as long-term load redistributions. We propose a framework for investigating application-specific index structures to create locality-aware partitioning schemes (so-called histograms) and to find appropriate data mapping strategies. We particularly investigate how far mapping strategies based on space filling curves preserve query locality and achieve data load balancing depending on query patterns in comparison to a random mapping. An efficient data dissemination technique for the anticipated large data volumes is important for several use cases within scientific federations, including initial data distribution and data replication. A scalable solution should neither induce a high load on the transmitting servers nor create a high messaging overhead. Optimizing data distribution with regards to latency and bandwidth is infeasible in our scenario. Therefore, we propose several strategies that optimize network traffic, use chunk-based feeding, and improve data processing at receiving nodes in order to speed up data feeding. In the face of different typical submission scenarios, we show how community-driven data grids can adapt their query coordination strategies during query processing. We explore the impact of uniform of skewed submission patterns and compare multiple strategies with regards to their usability and scalability for data-intensive applications. Our techniques improve query throughput considerably by increased parallelism and data load balancing in both local as well as wide area deployments. Addressing skewed query workloads, so-called query hot spots, by query load balancing and directly meet the requirements of a data-intensive e-science environment is another interesting and challenging task. We enhance our data-driven partitioning schemes to trade off data load balancing against handling query hot spots via splitting and replication. We use a cost-based approach for workload-aware data partitioning. Based on these workload-aware partitioning schemes, we use master-slave replication to compensate for short-term peaks in query load and address long-term shifts in data and query distributions by partitioning scheme evolution. Our research prototype HiSbase realizes the concepts described within this thesis and offers a basis for further research shaping the data management of future scientific communities.